Forum on Public Policy 


Criterion-Referenced Assessment for Language Organization: An Example 
of Evidence-Based Practice 

Joan S. Klecan-Aker and Karen Colson 

Joan S. Klecan-Aker, Ph.D., CCC-SP, Professor, Department of Communication Sciences and Disorders, Texas 
Christian University 

Karen Colson, Ph.D., Assistant Professor, Baylor University 

Abstract 

Reliable criterion-referenced assessments are critical as one of the first steps in evidence-based practice. These 
assessments must also be valid. One of the most important skills to measure in school-age children with language 
disabilities is the ability to organize language in the form of narratives. 

The purpose of this paper is to describe a criterion-referenced assessment as a baseline measure for 
improving language organization (narrative ability) in children with language disorders. Descriptions of elicitation 
techniques, transcription methodologies, story grammar components, developmental levels, cohesive ties and 
referencing are included as examples of different ways to measure narrative ability. 


Introduction 

There are two types of test instruments that are used to assess children with language disabilities. 
These instruments include standardized tests and criterion-referenced measures. Each of these 
assessment methodologies has strengths and weaknesses. Standardized tests are useful in 
providing a diagnosis. In addition, they tend to be reliable indicators of a child’s ability as a 
result of that standardization. However, standardized assessments cannot provide targets for 
intervention. They can “lead the way” because there may be certain skills that are identified as 
problem areas, but, typically, those areas are not addressed in depth in the test. For example, a 
child may miss one or two items that assess subject-verb agreement. However, two items are not 
enough data to draw the conclusion that subject-verb agreement is, indeed, a global deficit. 
Criterion-referenced assessments are needed for that task. A criterion-referenced assessment 
compares a child to a preset criterion, unlike standardized tests that compare a child’s 
performance with other children of the same age or grade. Criterion-referenced assessments also 
are considered to be more valid indicators of a child’s performance. For example, if a child was 
diagnosed as having difficulty with subject-verb agreement, a clinician would design a criterion- 
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referenced assessment to test that skill in depth. In other words, standardized tests are good for 
obtaining a diagnosis while criterion-referenced assessments are better for selecting targets for 
intervention. 

Evidence-Based Practice 

Recently, a need for evidence-based practice has become very important as a means of 
showing that intervention is successful. Evidence-based practice has been defined as “the 
integration of best research evidence with clinical expertise and patient values” (Sackett, Straus, 
Richardson, Reosenberg, & Haynes, 2000, p. 1). The American, Speech, Language and Hearing 
Association (2005) also cites this definition and defines components that can aid clinicians in 
developing evidence-based procedures in their intervention programs. 

Nelson (2010) suggests that evidence -based practice involves a five-step process. The 
steps are: 1) ask a question that is relevant to meeting a particular client’s needs, 2) search for 
available evidence, 3) critique the quality of the evidence, 4) apply the evidence to one’s own 
practice and 5) evaluate effectiveness in terms of outcomes for that particular client. An example 
of asking a question that is relevant to meeting a particular client’s needs begins with the 
diagnostic process and segues into a criterion-referenced assessment if that assessment was not 
part of the original diagnostic procedure. For instance, a child may have difficulty organizing the 
language or telling narratives. The next step, then, would be to determine what intervention 
procedures are best to achieve a desirable outcome for the client. At this point, the clinician 
must turn to the research literature. A textbook is not a good place to go for this information. 
Instead, the clinician needs to access search engines for empirical evidence that certain 
methodologies are, in fact, effective and examples of evidence-based practice. Cochrane 
Collaboration (www.cochrane.org/reviews), or the What Works Clearinghouse 
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( w w w ■ whatworks . ed . gov ) are good places to start. Also, professional organizations such as the 
American Speech-Language-Hearing Association provide tools for finding examples of effective 
intervention (Nelson 2010). 

As the clinician peruses the research, it is important to consider the participants or clients 
that were selected, the nature of the treatment and the type of research design. At this point, the 
clinician then should have an idea of what methodologies have been used successfully and if 
those methodologies are appropriate for the client. Once an intervention approach is selected, 
the clinician then determines a method of data collection. Data collection usually begins with a 
criterion-referenced assessment. Some professionals refer to this assessment as establishing 
baselines or probing. 

Establishing Baselines 

Baseline probes should be taken prior to the implementation of treatment. In single- 
subject designs, two baseline methods have been used quite successfully. The first design is 
called ABAB. One condition is alternated with the other. A stands for documenting a stable 
baseline. There is no intervention during this phase. For example, if the goal is to improve a 
child’s ability to organize the language in story form, the clinician could elicit spontaneous 
narratives prior to the beginning of treatment. The baseline must be stable, so more than one 
narrative should be elicited. The next step, represented by B is the implementation of 
intervention. At the end of the therapy program, the clinician takes data again in the same way 
as data were collected prior to treatment. The fourth step would be intervention for the second 
period. If treatment is effective, the child’s skill in organizing the language in story form should 
be indicated by a change in the data after treatment. 
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This ABAB design is sometimes called a time-series design. Rather than use a single 
pretest and a single posttest, several baselines are established prior to and after treatment. As 
before, this design starts with establishing a baseline before the experimental treatment begins. 
The difference is that several baselines may be established, so instead of one or two stories being 
elicited, the clinician might have the child tell four stories one week apart. This ensures that the 
original baseline is very stable. The strength of this design centers on the fact that the repeated 
measurements of the child’s ability to tell a story provides relatively good control over threats to 
internal validity such as maturation, etc. In a sense, clients can act as their own control group 
during the baseline segment of the design, because the clinician can examine the child’s ability 
on the repeated measurements without the intervention. Another advantage of this design is that 
it lends itself to school schedules. Baselines are established at the beginning of the school year, 
at Christmas break and at the end of the school year. Multiple measures can be taken at each 
juncture with therapy occurring during the school year. 

The Importance of Narratives as a Measure of Language Organization 

Children use narrative language to create stories and descriptions of events in their daily 
lives. Comprehending and producing narratives require that children use several skills at once. 
Thy must coordinate phonologic, morphologic, syntactic, semantic, and pragmatic knowledge. 
By the end of the elementary school years, children can be expected to produce richly detailed, 
multiple-episode stories that contain complex sentences. 

There are several reasons why researchers and clinicians have found the examination of 
children’s narratives useful. First, nearly every child can respond to the task of being asked to 
tell a story about something. Additionally, having a child tell a story provides the clinician with 
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an uninterrupted flow of discourse from the child; thereby avoiding certain artificialities of data 
from conventional elicitation. In addition, we know that children with language impairments tend 
to produce narratives with fewer total words, fewer different words, more syntactic errors, poorer 
use of cohesive devices and less story grammar content (Swanson, Fey, Mills & Hood (2005). 
Fey and colleagues also found that children with language impairment in kindergarten who 
appeared to have recovered by second grade more closely resembled their typically developing 
peers than children with persistent language difficulties with narrative type tasks. What is 
especially interesting, however, is that by fourth grade, this same group of children exhibited 
profiles of narrative skills that made them look more like children with persistent language 
problems. The authors concluded that, regardless of what seems to be like recovery in second 
grade, school-age children with early histories of language impairment may need intervention 
focusing directly on their narrative abilities (2004). Finally, we know that children with 
language disorders frequently have difficulty comprehending and producing narratives, and this 
can negatively affect their intellectual, social, and academic development (Gillam & Pearson 
2004). 

Elicitation of Narratives and Transcription Procedures 

There a number of ways to elicit and analyze narratives, but the one described in this 
paper was developed by Klecan-Aker and Brueggeman (1991). Many studies have been done 
since the procedure was developed, both in terms of determining the reliability of the procedure 
and the use of the data as a framework for intervention (Klecan-Aker 1993; Klecan-Aker, 
Flahive, & Fleming, 1997; Klecan-Aker & Gill, 2005). 
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In their procedure (1991), a minimum of two narratives are always elicited. The reason 
for eliciting more than one is to ensure that the baseline is stable. If the clinician and client have 
the time, the elicitation of one or two more narratives should result in even a more stable 
baseline. Narratives are elicited by first providing the child with the model of what a story is. 
The clinician shows the child a picture and then generates a story about the picture. The reason 
for providing a model is that research has shown that just showing a child a picture and asking 
him/her to tell a story is not sufficient. (Klecan- Aker, Mclngvale & Swank, 1987). Children will 
label or describe the picture. By providing a model, the child understands the type of task that is 
being requested. All narratives are audiotaped and then transcribed. When transcribing, the 
stories need to be triple-spaced and written as one run-on sentence with no capital letters or 
punctuation. 

Story Analysis 

After the story is transcribed, it will be divided into t-units. A t-unit is a simple sentence 
or a complex sentence. It’s never a compound sentence because that would be the equivalent of 
two t-units (2 simple sentences). After the story has been divided into t-units, words/t-unit, 
words/clause and clauses/t-unit are calculated. Then, each t-unit is assigned a story grammar 
component. This component shows the role of the t-unit in the story. An analysis of story 
grammar is the next area of analysis. The type of story grammar components that are found in 
the story determine the story’s developmental level. The developmental level of both stories 
determines the starting point of intervention. For example, if children are telling level 2 stories, 
the clinician might decide to begin intervention with level 3 or level 4 stories. Story grammar 
components include the following: 
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1) setting statement- who the story is about and when and where the story takes place 

2) initiating event-the problem or the main point of the story 

3) internal responses-the reaction of the main character to the initiating event (thinking 
and feeling statements) 

4) action- an attempt to solve the problem 

5) consequence-the result of the action 

6) dialogue- asking or telling statements (they don’t need to be direct quotations) 

7) ending-the tells the final resolution of the story 

There are also seven developmental levels of stories. Level 1 stories occur when a child 
simply talks randomly. Level 2 stories occur when a child labels or describes items in the 
picture. Level 3 stories have the story core of initiating event, action and consequence. Level 4, 
5, 6 and 7 happen which children add additional story grammar components, one for each level. 
The reason why the component is not specified is because different children add different 
components. Research has not revealed any specific pattern. Level 0 stories are stories that 
don’t fit into any other category (Caraway & Klecan-Aker, 1994). 

Types of Baseline Data Collected from the Stories 

In addition to the t-unit analysis, assignment of story grammar components and the 
determination of developmental level, other analyses can be completed as well. The most 
common analysis is an assessment of cohesive ties, specifically the use of conjunctions to tie 
clauses together and the use of references. Conjunctions are examined in terms of how they are 
used to tie clauses together. The number and type of conjunction is tallied. For example, 
conjunctions can either be coordinating or subordinating. Research suggests that the ratio of 
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coordinating to subordinating should be approximately 4:1. References are either appropriate or 
inappropriate when used as a noun substitution for subjects and objects. Nominative or subject 
pronouns include I, you, he, she, it, we and they. In contrast, objective pronouns are those 
pronouns that are used when the pronoun serves as the object. They are me, you, him, her, it, us 
and them. 

Consider the following examples in which the personal pronouns are in bold. 

1) He will not leave. 

2) Lisa told Susan that she doesn’t want to go. 

3) Jack hit her on the head. 

4) Why won’t Susan talk to him? 

In these examples, the pronouns are serving as the doers of the action. In the latter two 
examples, the objective pronouns are used because they are serving as the recipients of the action 
(Justice & Ezell, 2002). Once a noun has been stated, a child may use a pronoun substitution for 
two additional sentences or clauses. After that point, the noun needs to be restated or the 
pronoun is said to be inappropriate. Another way to determine inappropriate is that if the reader 
cannot determine who or what the pronoun is referring to. 

In conclusion, narrative elicitation and analysis yield a variety of possible intervention 
targets. These targets include: 

1) increasing the use of all the story grammar components 

2) increasing the developmental level of the stories 

3) improving the use of subordination to add complexity to the children’s stories 

4) improving the use of appropriate references 

5) decreasing words per t-unit 
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6) increasing clauses per t-unit. 

Because the use of narratives is a criterion-referenced measure, there are no 
developmental norms. However, we know that the narrative genre presented in this paper is 
similar to the genre used in children’s academic material in grades kindergarten through third. 

Summary and Conclusions 

Children’s narratives provide a rich source of objective data from which a variety of 
treatment targets can be selected. This information is then used as baseline data and, therefore, 
as a way of measuring progress in therapy. As mentioned earlier, the school setting provides an 
excellent time-table for using this type of criterion -referenced measure. Also, it’s important to 
understand that the best way to measure progress in therapy is not only to calculate change from 
a specific target such as improving the use of appropriate references, but to also elicit the 
complete narrative again. By doing that, the clinician can see if what she or he has taught has 
generalized to the stories themselves. 

Given that research has shown that improving narrative ability is an important skill for 
pre-school and school-age children, teaching individuals with language disorders how to 
organize language in story form is a reasonable intervention objective. Children tell personal 
narratives about the experiences that shape their lives during conversations with friends, parents, 
and teachers. Often, these personal narratives fall apart when the children with language 
disabilities attempt to tell stories about their experience. Because narrative ability is so 
important, it is critical that reliable baselines are established before one can state that the 
intervention program used to improve narrative ability is a good example of evidence -based 
practice. 
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